In an era of rapid development in artificial intelligence, the intelligence level of large models continues to improve, but the challenges of inference system efficiency are becoming increasingly apparent. Addressing high inference loads, reducing inference costs, and shortening response times has become an important issue faced by the industry. Kimi has collaborated with the MADSys laboratory at Tsinghua University to launch the KVCache-based Mooncake inference system design plan, which will be officially released in June 2024. The Mooncake inference system enhances efficiency through its innovative approach.

Recently, AI chip company Groq, based in California, has reached a partnership agreement with Saudi Arabia's oil giant Aramco to establish a large data center in Saudi Arabia. The aim of this data center is to enhance Saudi Arabia's capabilities in the field of artificial intelligence and further promote the diversification of the country's economy. Groq's CEO, Jonathan Ross, stated that the data center is expected to be operational by the end of the year, initially equipped with 19,000 Language Processing Units (LPUs). Image source note: Picture

This article outlines the latest developments in the AI inference chip field, particularly the results of the MLPerf Inference v4.1 competition, which gathered the latest chips from companies such as AMD, Google, UntetherAI, Cerebras, FuriosaAI, and Nvidia. Nvidia's H200 and GH200 chips maintain a lead in the 'Data Center Closed' category, but chips from other companies excelled in specific tasks, especially in terms of energy efficiency. AMD's Instinct Accelerator.

Cerebras Systems Inc. announced the launch of what it claims to be the 'world's fastest' AI inference service, challenging Nvidia Corp. The service aims to perform AI tasks at greater speed and lower cost. It is based on the WSE-3 processor, which has over 900,000 computing cores and 44GB of memory, with a core count 52 times that of Nvidia's H100. It claims an inference speed of up to 1,000 tokens per second, which is 20 times faster than Nvidia's comparable service, and offers a price-performance ratio 100 times better. The service is offered for free.

Cerebras Systems has launched Cerebras Inference, claiming it to be the fastest AI inference service in the world, achieving performance that surpasses traditional GPU-based systems by 20 times with significantly improved cost-effectiveness, particularly suitable for processing large language models (LLMs). Its 8B version processes 1800 tokens per second, while the 70B version processes 450 tokens, with speed and cost-performance far exceeding NVIDIA GPU solutions.

Meta engineers estimate that next year, there will be an addition of 1 to 2 million Nvidia H100 graphics processors for global AI inference. If all of them are used for language model generation, the energy consumption would fall within a reasonable range, generating 100,000 tokens per person per day. Only two additional nuclear power plants would be required to provide sufficient power for global AI inference. Model training faces limitations due to data scarcity, and researchers are working to improve model efficiency. In the next 3-4 years, it is expected that we will know whether current technology can achieve general AI.